Papers with training scheme
Uncertainty Modeling for Machine Comprehension Systems using Efficient Bayesian Neural Networks (2020.coling-industry)
Copied to clipboard
| Challenge: | Neural approaches have improved machine comprehension tasks, but models often operate as a black-box, resulting in lower interpretability. |
| Approach: | They propose a hybrid approach to quantify model uncertainty using Bayesian weight approximation and boost up inference speed by 80% relative to test time. |
| Outcome: | The proposed approach boosts inference speed by 80% relative to the previous approach and is applied to a clinical dialogue comprehension task. |
Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn mappings in the embedding space of text autoencoders are limited to a single-vector embeddment, which limits how much information can be retained. |
| Approach: | They propose a method to learn mappings in the embedding space of an autoencoder by extending it to Bag-of-Vectors Autoencodeurs (BoV-AEs) this allows to encode and reconstruct much longer texts than standard autoencodings . |
| Outcome: | The proposed method performs better than a standard autoencoder on unsupervised sentiment transfer. |
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one. |
| Approach: | They propose a framework to assess adversarial robustness of multiple KD algorithms. |
| Outcome: | The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods. |
Simple and Effective Multi-Paragraph Reading Comprehension (P18-1)
Copied to clipboard
| Challenge: | Existing question answering models cannot scale beyond short paragraphs, so adapting a model to document-level input is difficult. |
| Approach: | They propose a method of adapting neural paragraph-level question answering models to document input. |
| Outcome: | The proposed method achieves state-of-the-art on TriviaQA and SQuAD and a 10 point gain on SQuADA. |
RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) achieve high recall amongst top few predictions, but low overall accuracy, motivating the need for answer re-ranking. |
| Approach: | They propose a method to make answer re-ranking successful for span-extraction tasks even beyond large pre-training. |
| Outcome: | The proposed approach achieves 45.5% Exact Match accuracy on Natural Questions and 61.7% on TriviaQA. |
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
Playing the Part of the Sharp Bully: Generating Adversarial Examples for Implicit Hate Speech Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing algorithms for hate speech detection focus on explicit forms of hate speech, but they fail to properly detect subtle and implicit HS messages. |
| Approach: | They propose a framework for generating adversarial implicit HS short-text messages using Auto-regressive language models and a strategy to group the generated messages in complexity levels. |
| Outcome: | The proposed framework shows that iteratively retraining on HARD messages significantly improves implicit HS benchmarks. |
LeeBERT: Learned Early Exit for BERT with cross-level optimization (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are resource exhaustive and computationally expensive for industrial scenarios. |
| Approach: | They propose a learning scheme to learn from each other to speed up inference . they ask each exit to learn the weights of different loss terms, instead of learning only from the last layer . |
| Outcome: | The proposed scheme improves state-of-the-art (SOTA) early exit methods for pre-trained models on the GLUE benchmark. |
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (D19-1)
Copied to clipboard
| Challenge: | Detecting offensive language in the context of a dialogue is an increasingly important application of natural language processing. |
| Approach: | They propose to train a model to be robust to such attacks by iterative build it, break it, fix it scheme with humans and models in the loop. |
| Outcome: | The proposed model is significantly more robust to such human attacks than previous systems. |
Towards Label-Agnostic Emotion Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing representation schemes for emotion analysis are based on label formats, natural languages, and even disparate model architectures. |
| Approach: | They propose a training scheme that learns a shared latent representation of emotion independent from different label formats, natural languages, and even disparate model architectures. |
| Outcome: | The proposed model performs well on a wide range of datasets without penalizing prediction quality. |
Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that translation quality alone is not sufficient for measuring knowledge transfer in multilingual neural machine translation. |
| Approach: | They propose a method that measures representational similarities between languages to measure knowledge transfer. |
| Outcome: | The proposed method improves translation quality for low- and mid-resource languages across multiple datasets and models. |
ISR: Self-Refining Referring Expressions for Entity Grounding (2025.acl-long)
Copied to clipboard
| Challenge: | Entity grounding is a crucial task in the construction of multimodal knowledge graphs. |
| Approach: | They propose a novel scheme to enhance the multimodal large language model's capability to generate high quality REs for the given entities as explicit contextual clues. |
| Outcome: | The proposed method surpasses other methods in entity grounding, highlighting its effectiveness, robustness and potential for broader applications. |